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I. Introduction 

Incomplete: How Middle Class Schools Aren’t Making the Grade is a new report from Third 
Way, aWashington, D.C.-based policy think tank 1 The report aims to convince parents, 
taxpayers and policymakers that they should be as concerned about middle- class schools not 
making the grade as they are about the failures of the nation’s large, poor, urban school districts. 
The report, authored by Tess Stovall and Deirdre Dolan, attempts to raise alarms for the average 
parent, taxpayer, and policymaker who, as the report argues, think that their local public schools 
are a lot better than they really are. 

The intent of the report, as stated in its concluding section, is to “ignite the conversation about 
middle-class schools.” While not offering specific policy recommendations in the current report, 
the authors promise that 

In future papers, we will explore policy solutions that will directly benefit middle- class 
schools. These policy solutions, no doubt, could also help upper and lower- income schools, 
but it’s important to recognize that education reforms directed toward improving middle- 
class schools are needed (p. 15). 

While it may in fact be true that greater policy emphasis should be placed on improving middle- 
class schools, the present report provides no reliable analysis or valid basis for such a shift in 
priorities. 



II. Findings and Conclusions of the Report 

The authors contend that while the majority of American school-aged children are served in 
middle- class public schools, these schools have fewer resources and perform much less well than 
the public expects or assumes. The report argues for greater policy emphasis on schools it 
designates as “middle class.” 

Specifically, the authors maintain that middle- class schools serve 53% of the U.S. student 
population (p. 2), that teachers in middle-class schools are paid less than their counterparts in 
higher- or lower- income schools (p. 3), that pupil- to- teacher ratios are higher in middle-class 
schools than in higher- or lower- income schools, and that per- pupil spending in middle- class 
schools is lower than in the other two groups (pp. 2-3). 
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The authors’ primary emphasis is on their findings that while the parents and taxpayers have 
high expectations for their middle- class schools, and while graduates of middle- class schools are 
the “backbone of the U.S. economy” (p. 1), “Only one in four middle-class high school graduates 
will obtain a college degree by age 26” (p. 6). 



III. Report’s Rationale for its Findings and Conclusions 

The main conclusions of the report center around claims that “middle-class” schools perform 
much less well than people— the parents, taxpayers, and policymakers— think they do, and that 
these people should be outraged by the performance of these schools. To support its 
conclusions, the report links to a series of tables from compendiums prepared by the National 
Center for Education Statistics (NCES) and to secondary sources using NCES data. Summary 
tables of the student outcome data purported to back its main conclusions are presented in the 
body of the report on pages 11 and 12 of section three, and in Appendix II. 

Setting aside methodological concerns regarding the definition of “middle class,” addressed in a 
subseguent section of this review, the biggest problem with the report is that even the data that 
are cited fail to make the case that “middle-class” schools perform less well than expected. 

There is no clear statement of what the expected performance of middle-class schools should be. 
Rather, the report includes a citation to the Phi Delta Kappan national public poll, showing that 
parents often rate their local school as an “A” or a “B” on a typical grading scale. 2 This polling 



The data that are cited fail to make the case that “middle-class” schools 
perform less well than expected. 



result is used to imply a “high” expectation among parents and taxpayers, and it is also used to 
imply specifically that this expectation applies to the report’s classification of “middle-class” 
schools. The authors then take out of context National Assessment of Educational Progress 
(NAEP) proficiency rates and four- year college attendance and completion rates, using heavily 
bias- laden statements, again to encourage outrage. For example, the header of the executive 
summary states: “If you discovered that only one in four graduates from your neighborhood 
high school would earn a college degree, would you be alarmed?” (p. 1). Similarly, one large font, 
bold highlighted conclusion states: “Only one in four middle- class high school graduates will 
obtain a college degree by age 26” (p. 6). 

The authors should be praised for thoroughly citing their data sources and for providing clear 
summary tables. But a close look at these citations and tables shows that their own data does not 
support their conclusions. For example, the report’s “educational outcomes” table (p. 12), 
reproduced below, shows that what the authors refer to as “middle-class” districts actually 
perform right where they would be expected to by researchers who have studied the link 
between family income and student outcomes: solidly between lower- and higher- income 
districts. The same is true in every table and analysis provided in the report. Middle- class 
schools invariably fall, in the middle. 3 
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Table l. Educational Outcomes 





Upper Income 


Middle Income 


Lower Income 


12 tn Graders that 
Graduate High School 


91% 


84% 


68% 


High School 
Graduates that 
Immediately Attend a 
4-year College 


52% 


38% 


29% 


High School 


47% 


28% 


17% 



Graduates that 
Obtain a College 
Degree by Age 26 

Reproduced from unnumbered table, “Educational Outcomes,” in Stovall, T. & Dolan, D. ( 2011 ). Incomplete: How 
Middle Class Schools Aren’t Making the Grade. Washington, DC: Third Way, 12. Retrieved September 19, 20 11, from 
http:/ / thirdway.org/ publications/ 435 

In addition, boldly highlighted statements and section headings freguently do not match the 
data. This problem is even found in the report’s Appendix II, where the authors state 

the middle- class schools— highlighted in yellow— do not perform at the levels that parents 
and taxpayers think they do, but there has been little time, energy or focus paid on 
improving the achievement of these schools (p. 17) . 

As noted, the trouble with this statement, as with the above conclusions, is that the data in the 
report’s own table— a table that immediately follows the statement and conclusions— show that 
“middle-class” schools perform right in the middle, between upper- and low- income schools. 
There is no basis whatsoever provided for the claim that somehow taxpayers “think” middle-class 
schools perform better than this. The polling data showing that parents tend to give their local 
schools high grades is a phenomenon that readers of the Kappan have found interesting over the 
years, but is not evidence that parents and taxpayers think that middle- class schools do better 
than schools serving upper- income students. No evidence is provided to demonstrate such a view. 

The report also draws conclusions about the resources available to schools serving different 
income categories. The information presented includes per- pupil spending, 4 teacher salaries 5 
and pupil- to- teacher ratios. 6 In each case, the report points out that middle- class schools have 
less (than higher- income or lower- income schools). The report does not, however, make specific 
policy recommendations about targeting more resources to the middle class. Such 
recommendations are perhaps implicit. In any case, as will be discussed in the next section, all 
of the resource comparisons in the report are based on relatively meaningless national averages, 
without any consideration for regional cost variation and other important contextual factors. 

IV. Report’s use of the Research Literature 

The summaries of literature presented in sections of the report such as “The Economic 
Imperative” (p. 3) and “The National Impact” (p. 13) are neither balanced nor comprehensive, 
and the literature- based claims made are largely speculative. 
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The report makes economic impact claims based on speculative interpretations of non- peer- 
reviewed policy reports. The report claims, for example, that if the “U.S. had improved the 
overall student achievement levels to those of Finland and Korea, the Gross Domestic Product of 
the U.S. would have been g% to 16% higher in 2008 than it was” (cited at fn. 72 to the 2009 
McKinsey report The Economic Impact of the Achievement Gap in America’s Schools. 7 ). A 
similar claim, that “20 million post- secondary students over the next fifteen years will add $500 
billion to the Gross Domestic Product and increase wages for all workers, even those with only a 
high school diploma” (p. 3), cites for support a non-peer reviewed report from the Georgetown 
University Center on Education and the Workforce . 8 

The report also dtes a handful of other sources to contend that, if individuals attain higher levels 
of education, there will be benefits in wages to those individuals as well as substantial benefits to 
the economy. While one could easily guestion many of the assumptions underlying the various 
dted sources, espedally the more speculative projedions, these concerns pale by comparison to 
the severe methodological problems in the Third Way report. 

V. Review of the Report’s Methods 

The crux of the problems with the report’s methods is the overly broad and unchecked definition 
of “middle-class” schools. The report explains: 

Schools with 25% (or less) of students eligible for free or reduced- price lunches are 
considered “wealthy” or upper-income schools. Those with greater than 75% partidpation 
are deemed “lower-income” schools. Those with between 26% and 75% eligibility represent 
our target middle- class schools, (p. 2) 

Mixing and Matching Data Sources 

The Third Way authors rely on several data sources and, to their credit, document them clearly. 
Most of the authors’ data sources originate within the National Center for Education Statistics 
Common Core of Data or the National Center for Education Statistics Schools and Staffing 
Survey. In many cases, the authors construd their calculations of charaderistics of middle- class 
schools by combining the charaderistics of schools or distrids reported in tables from NCES 
reports as having between 25% and 75% children gualifying for free or reduced- price lunch. 

That is, the authors don’t actually analyze the district- or school- level data themselves, but 
rather aggregate information from tables already created by NCES. 9 The authors indicate that 
fmandal calculations were based on data from the New America Foundation, which compiles a 
national data set of distrid charaderistics, using finandal data and enrollment data from NCES, 
and adding assessment data from state data systems. 

The authors seem to have overlooked the fad that NCES tables based on Schools and Staffing 
Survey data typically report charaderistics based on school- level subsidized lunch rates. As 
such, within a large, relatively diverse distrid like New York City, several schools would fall into 
the authors’ middle-class grouping, while others would be considered high- poverty, or low- 
income, schools. But, many other of the authors’ calculations are based on distrid- level data. 
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such as the financial data from New America Foundation. When using district- level data, a 
whole district would be included or excluded from the group based on the district- wide 
percentage of children gualifying for free or reduced- price lunch. What this means is that the 
Third Way report is actually comparing different groups of schools and districts from one 
analysis to another, and within individual analyses. 

As noted previously, the authors’ main conclusions focus on the alleged failure of middle- class 
schools to produce college graduates. While most of their analyses rely on relatively simple 
tabulations of NCES data, a far more convoluted approach is used for determining college 
completion. For this analysis, as explained in their lengthy footnote #90 (p. 31), the authors rely 
on completion rate summary tables from William Bowen’s book, Crossing the Finish Line. 10 
Unfortunately, most analyses in Crossing the Finish Line rely on databases of students 
attending 21 selective public flagship universities across the country, and a broader set of public 
universities in four states (Maryland, North Carolina, Ohio, and Virginia). In other words, 
Bowen does not provide a nationally representative sample, either of students attending college 
or of high school graduates. 11 Further complicating matters, Bowen and colleagues report 
completion patterns by individual students’ family income quartiles, not school or district 
subsidized lunch rates. 12 

While not entirely clear, it appears that the Third Way authors may have relied primarily on 
completion rate data presented in Chapter 2 of the Bowen book (given that the only page 
citation in footnote #90 provided is to Page 21) . While these particular tables are drawn from a 
nationally representative sample, 13 that sample is of the high school graduating class of 1992 
and, as in their other analyses, the quartiles in the analysis are by family income (as well as 
parent education level). 

Essentially, the Third Way authors are asserting that the two middle- income quartiles of 
students in these tables are representative of all students, nationwide, who currently graduate 
from middle- class school districts enrolling 26% to 75% free or reduced- price- lunch students. 
Readers cannot, however, know whether these middle- income students attended middle-, 
upper-, or lower- income schools or districts. This is an unfounded and irresponsible assertion. 

Defining Middle Class 

A much bigger problem, and probably the most significant problem throughout this report, is 
the choice to define as “middle class” those schools or districts with 26% to 75% of children 
qualifying for free or reduced- price lunch (that is, those children whose families fall below the 
185% income threshold for poverty). In the authors’ defense, they seem to have gotten this idea 
from one of the NCES reports they dte, the 20 10 Condition of Education, special section on 
high- poverty schools, which notes: 

Throughout the special section, high- poverty schools are compared with low- poverty 
schools. In order to cover the breadth of material in the limited space of this special section, 
the middle two FRPL quarters (26-50 and 51-75 percent) are not usually discussed. 14 
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The Third Way authors seem to have taken this statement as a challenge to go where no 
researcher had gone before, not realizing that it was in fact a challenge to go where no 
researcher should go ever. It is one thing to isolate the highest and lowest “quarters” ; it is quite 
another to define all schools between them as “middle class.” Why this is a poor choice is 
illustrated later in this section of the review. But here’s a teaser question to keep in mind: Do 
you consider Detroit to be middle class? 

The Third Way report implies throughout that the intent is to characterize “middle-class” 
schools, nodding toward the public perception of “middle class.” Here’s the opening of the 
report’s Executive Summary (internal citation omitted): 

For decades, there has been a laser- like focus in education reform on the lowest- performing 
students and schools. This focus continues to be critical for maintaining America’s social 
fabric and ensuring that all children have an opportunity to succeed, but it is not enough. In 
this paper, we urge that America must embark upon a second phase of education reform that 
squarely focuses on dramatically improving achievement in the middle- class schools that the 
majority of children attend. 

Our findings show that middle- class schools seem to be forgotten in the education debate. 
There is a paucity of academic literature on their performance, expectations, and on ideas for 
reform. Yet, they produce the students who are the backbone of the U.S. economy. Among 
parents of school-aged kids in middle- class jurisdictions, there is a strong belief that these 
schools are educating students at the highest levels. More than seven of ten parents with 
children in the public schools grade their kids’ schools as either an A or a B, and nine of ten 
parents of school-age children expect their kids to go to college. But that is far from the 
reality. Middle- class schools are falling short on their most basic 21 st century mission: to 
prepare kids to get a college degree. 

In order to maintain a prosperous middle class, grow our economy, and foster a public 
education system that taxpayers deserve, it is necessary to shine a light on the experience of 
middle-class students. These are students that don’t attend America’s best schools but also 
don’t attend the worst. They attend the schools that are in every city, town, and suburb. For 
our nation to succeed, their schools must be college factories— graduating high school 
students who are prepared to get to and through college (p. 2). 

The problem is that classifying either schools or districts according to the chosen parameters 
does not, in fact, capture a set of schools or districts that one would generally associate with 
“middle class.” The schools and districts included in the definition are, therefore, hardly the 
ones “forgotten in the education debate.” 

For example, below is a list of school districts that, according to the same 2008-2009 New 
America Foundation data used by Third Way, are (a) large central city districts, (b) with over 
50,000 students, (c) with greater than 50% and but less than 75 % of children qualifying for free 
or reduced- price lunch. 15 That is, these large urban districts are counted in any Third Way 
district-level analyses as “middle-class” districts. They include the urban districts of Detroit, 
Houston, Chicago, New York, Los Angeles, Baltimore, Memphis and Philadelphia. That is, this 
report defines some of the poorest large-city districts in the country as “middle class.” 
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Table 2. Larger Urban Districts in the Mid-Range 
for Free and Reduce-Price Lunch 



District 


% Free or Reduced-Price Lunch 


Albuquerque Public Schools 


51.8% 


Arlington Independent School 
District 


54.7% 


Austin Independent School District 


62.4% 


Baltimore City Public School System 


73.1% 


Boston 


74.3% 


City Of Chicago School District 2 


73.4% 


Columbus City School District 


70.2% 


Denver County 1 


65.9% 


Detroit City School District 


74.3% 


El Paso Independent School District 


68.3% 


Fort Worth Independent School 
District 


71.7% 


Houston Independent School District 


63.5% 


Long Beach Unified 


68.2% 


Los Angeles Unified 


74.6% 


Memphis City School District 


68.9% 


Nashville-Davidson County School 


65.1% 


New York City Public Schools 


72.1% 


Philadelphia City School District 


73.2% 


San Antonio Independent School 
District 


55.3% 


San Diego City Unified 


63.2% 


San Francisco Unified 


55.5% 


Tucson Unified District 


52.7% 



Tabulation based on data from http:// febp.newamerica.net/ data/kl2_ district-data- 16.csv, 

Retrieved September 19, 2011. 

Yet the report never acknowledges or confronts this tension between the definition and the 
realities as understood by the “parents, taxpayers, and policymakers” cited as a touchstone 
throughout the report. It is guite likely that few readers of the report would consider these districts 
to be the “middle-class” districts that the report is supposedly addressing. Perhaps more 
importantly, these are the very districts that policymakers from both parties are now focusing on 
as they tinker with No Child Left Behind. That is, while these districts may not benefit from the 
attention they receive, they are not in any way “forgotten in the education debate.” 

Figure 1, prepared for this review of the report, shows the national distribution of school 
districts by percentage of students gualifying for free or reduced- price lunch, with the red 
vertical lines showing the Third Way cut offs. Notably, just inside the area designated “middle 
class,” among higher poverty districts, are many of the large cities displayed above. These 
districts are on the margin of this very high- poverty threshold, and they serve very large 
numbers of poor children. By including these district among those analyzed as middle class, the 
report’s authors severely skewed any and every calculation presented in the report. Tossing truly 
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National Distribution of Districts 2009 

With Middle Class Schools Outpoints 

CO - 



C\J - 



0 .2 .4 .6 .8 1 

% Free or Reduced 2009 

Data Source: http://febp.newamerica.net/data/k12_district-data-16.csv 

weighted by district enrollment 



Figure l. Distribution of Districts, with Third Way Cutpoints 

middle- class districts in a pool along with large (and largely poor) urban districts tends to shift 
the results toward the latter. 



National Distribution of Districts 2009 

With 25%ile & 75%ile District Cutpoints - Pupil Weighted 

CO - 



C\J - 



0 .2 .4 .6 .8 1 

% Free or Reduced 2009 

Data Source: http://febp.newamerica.net/data/k12_district-data-16.csv 

weighted by district enrollment 



Figure 2. Weighted Quartile Cutpoints 
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Figure 2 shows where the cut points would have fallen had the authors simply used true 
quartiles instead of fixed rates of qualified children and had they weighted those quartiles by the 
numbers of children in each quartile. The middle two quartiles in this case, exclude the high 
poverty urban core districts. 

Ignoring Context and Failing to Adjust Data 

Although it would have been an improvement, even the approach shown in Figure 2 is far from 
sufficient for accurately identifying middle- class schools or districts. A major problem with 
using subsidized lunch income thresholds is that the thresholds are not adjusted for regional or 
local economic conditions. That is, the same income threshold to identify low- income students 
is used in eastern Tennessee and in New York City, regardless of substantial differences in the 

The report suffers from basic flaws related to the failure to adjust for 
different contexts. 

quality of living that can be afforded in each location at the same income level. This is partly why 
many large urban centers, which are in fact high- poverty areas, appear to have lower subsidized 
lunch rates than we might expect. The same is true when making comparisons between families 
living in metropolitan areas and those in rural areas, where similar incomes afford very different 
lifestyles. Identification of middle- class districts requires a far more nuanced analysis with 
consideration for geographic location both at the macro and micro level. 

More generally, the report suffers from basic flaws related to the failure to adjust for different 
contexts. For example, the finding that middle- class schools spend less is derived from data that 
are simply averaged (weighted for total enrollments) from Census Fiscal Survey (F33) data, 
provided through a secondary source (New America Foundation 16 ) . There is no attention paid to 
regional differences in the value of the education dollar, or differences in other major cost 
factors including economies of scale, population sparsity, and various student needs. 17 Similarly, 
the finding that “middle-class” schools pay their teachers less depends on salary data that are 
averaged without regard for regional variation in competitive wages. 18 If “middle-class” schools 
are more concentrated in some regions and some states than others (which they are), the 
average spending differences or average salary differences are as likely to be a function of those 
geographic differences as they are a function of being “middle class.” 



VI. Review of the Validity of the Findings and Conclusions 

Given the weakness of its analyses and the severity of its methodological flaws, valid conclusions 
cannot be drawn from the report. The report’s definition of “middle class” is exceedingly 
imprecise, systematically includes urban high- poverty schools, and is deceptively presented as 
aligned with common public perceptions of “middle class,” which is quite likely not the case. 
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Further, as shown above, even if we set aside the report’s severe methodological flaws, the 
conclusions that are drawn and boldly asserted in the report are often in direct conflict with data 
tables presented on the very same page. 

VII. Usefulness of the Report for Guidance of Policy and Practice 

This report provides no usable guidance for policy or practice. The concluding section includes a 
suggestion that the Third Way will follow up the report with a policy agenda for middle- class 
schools. Without a more accurate and precise definition of “middle class,” and without valid and 
relevant analysis of data, it is difficult to conceive of what that agenda might include. Would this 
be advice for helping schools in “middle-class” Detroit, the poorest city in the U.S .? 19 Or perhaps 
the advice would be for relatively wealthy communities like Memphis (seventh poorest) or 
Philadelphia (ninth poorest). 

In truth, the greatest value of this report for guiding policy and practice is that it reveals guite 
strikingly the problems associated with making vastiy oversimplified assumptions. The report is 
an extreme exemplar of bad analysis and even worse reporting, thus offering an effective 
teaching tool for use with graduate students, education reporters and policymakers. The report 
reveals the importance of validating and double- checking available data and measures to see 
that they are capturing what the authors intended— e.g., that the data-driven definition of the 
group to be studied actually aligns with the intended or perceived group to be studied. Finally, 
the report demonstrates why one should always make sure that assertions include in the text of 
a report are supported by data presented in its tables, or vice versa. 
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